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DETECTING MATERIAL FAILURES IN GROUND LOCATIONS 
FIELD OF THE INVENTION 

The present invention relates to remotely detecting material failures 
in a ground location by using an image sensor. 
5 BACKGROUND OF THE INVENTION 

The surveillance of ground topography is well known in the art. It 
is frequently the case that an aircraft or a satellite includes an image capture 
device such as a CCD. In ground surveillance it is highly desirable to detect 
whether there has been a material failure in a man-made object such as a road, a 
1 0 pipeline, an electrical grid, or other man-made structures of practical interest. 

When detected, a determination is made if remedial action must be taken. Often 
times a visual inspection of ground topography is provided by a land-based crew 
that traverses an area by vehicle or foot, to determine if there is a material failure. 
Airborne photographic systems can also be used for capturing images of adjacent 
1 5 areas. These images are then reviewed to determine if there is a material failure. 
SUMMARY OF THE INVENTION 
It is an object of the present invention to provide an improved way 
to automatically determine if there is a material failure in a man-made structure. 

This object is achieved by a method for capturing images of ground 
20 locations and for detecting the presence of material failures in man-made 
structures in such ground locations comprising the steps of: 

(a) providing an image sensor spaced remotely from the ground 
and which sequentially captures a number of images of various ground locations 
to provide digital images; 
25 (b) processing captured digital images to determine the 

presence of a potential material failure in a man-made structure in accordance 
with predetermined coordinate positions which locate the man-made structures in 
one or more of the captured digital images; and 

(c) indicating to a customer that a potential material failure has 
30 been detected in a predetermined coordinate position. 
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In many cases it is required to inspect man-made structures 
frequently in order to determine the likelihood or development of potential 
material failures. In many cases these inspections are done by a ground site 
survey; individuals visit these locations and take measurements or other form of 
5 data on-sight. This process becomes cumbersome, costly, inconvenient, and in 
many cases unreliable and unsafe due to the dangers present in remote locations 
and to potential false interpretations due to worker fatigue and other factors. 
Furthermore, remote locations are frequently in mountains, deserts and forests that 
are difficult to reach and frequent inspection requires the placement of permanent 

10 maintenance and inspection crews adding to overall costs. It is an advantage of 
the present invention to provide a more effective way of determining material 
failures in man-made structures by automatically processing images captured from 
a remote platform. This automatic processing can include comparing with 
previously detected images. This automatic processing can also include 

1 5 algorithms and expert systems that act in a predictive manner. 

A feature of the present invention is that a chemical agent or a 
chemical change associated with a material failure can be used in a man-made 
structure of practical interest that is particularly suitable for detection after a 
material failure. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 depicts a system for capturing images from an airborne or a 
satellite platform in accordance with the present invention; 

FIG. 2 is a flowchart in block diagram form of the process of 
capturing and processing images to detect potential material failures in man-made 
25 structures; 

FIG. 3 is a flowchart in block diagram form of an image processing 
algorithm which can be used in the system shown in FIG. 1 ; 

FIG. 4 illustrates a somewhat different embodiment of the present 
invention that uses a chemical agent to aid in detecting potential material failures 
30 in a man-made object; and 
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FIG. 5 illustrates the capturing of an image, analysis to identify a 
potential material failure, and communication over a channel to deliver 
information to and receive payment from a customer. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
5 A sensor system 1 employed in the capturing of images in order to 

identify material failures in man-made structures may be mounted on either an 
aerial or a satellite platform. Images of the ground containing various man-made 
structures such as roadways, pipelines, electrical power lines, agricultural, mining, 
real estate activity and the like are captured by this sensor system 1 . The term 

10 "man-made structure" can also include other human activities such as insecticide 
spraying which after application can be detected by sensor system 1. In such a 
case, an additive can be included in the insecticide spray that can be detected by 
the image sensor. Sequential images may be captured in digital form and either 
stored in the aerial or satellite platform to be transferred later or transmitted via a 

15 radio link to a control ground station. The capture device 2 includes an electronic 
sensor, typically a CCD or CMOS imaging array that along with some imaging 
optics captures a picture of the scene in electronic form. In some instances a 
special optical filter 3 is attached to the input to the CCD or CMOS detector to 
filter the light wavelengths which are incident upon the detector. This optical 

20 filter 3 is chosen so as to maximize the signal-to-noise ratio for the detection of a 
specific type of material failure. Alternatively, the ground location image can be 
captured by conventional photographic cameras. Film images would then have to 
be converted to digital images by an image scanner that includes an image sensor. 
The system 1 also has an image capture control circuit 4 that sequences the 

25 operation of the capture device 2. As will be clear from FIG. 1, the operation of 
the various elements shown in system 1 are under the control of a control 
computer 31 . The image capture control circuit 4 controls capture device 2 and 
sends position and orientation information to a position and orientation storage 
circuit 5 with each captured image. Position information in the form of spatial 

30 coordinates is provided by the customer in order to identify the location of man- 
made structures of interest. Such position information is also stored in position 
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and orientation storage circuit 5. Position and orientation data are used along with 
predetermined coordinate positions to locate the man-made structures in the 
captured image. Control computer 3 1 causes image data to be stored in image 
storage 6 and can be processed to identify features of a scene in image processing 
5 circuit 7. The processing sequence is also directed by control computer 3 1 of the 
image data in this instance is to enhance the capability of the system 1 to identify 
material failures in man-made structures. The image processing circuit 7 includes 
a storage memory (not shown) that includes a representation of different material 
failures to be detected and comparing the captured digital image with the material 

1 0 failures to determine the presence of a material failure, type of material failures 
and location of the material failures. With the exception of the capture device 2, 
the various elements of the system 1 maybe located either in the remote platform 
or at the ground station location. Moreover, many of the elements described can 
be embodied in software which can be understood to be within the control 

1 5 computer 3 1 . The capture device 2 is located in either the aerial or satellite 
platform or a fixed structure spaced above the ground. 

The overall process for detecting material failures in man-made 
structures is depicted in flowchart form in FIG. 2. The flowchart is in block 
diagram form and those skilled in the art will appreciate that many of the 

20 functions are controlled by the control computer 3 1 . The starting event includes 
initializing the capture device 2 and image storage 6 to erase any previously 
captured scene data. Next a new scene is captured in block 9 using the position 
information supplied by the customer to trigger recording of the images. The 
image data along with position and time information necessary to identify the 

25 location and time of the current scene is stored in order to facilitate comparison 
with the same scene taken at other times. Image and other data are stored in a 
scene database 10 in order to perform such comparisons at a future time. Image 
analysis 1 1 is next performed in order to identify changes in the scene and 
facilitate identification of material failures in the man-made structures that appear 

30 in the scene. The latest scene image is compared with image data that has been 
previously stored in the scene database 10. If a material failure is not detected the 



process stops. Detection of a material failure may initiate further image analysis 
12 as required by a customer 13. The identification process finishes with the 
results of the analysis communicated to the customer 13. The communication 
make take many forms, for example a telephone contact or e-mail notification of 
5 the detection of the material failure. The final step in the process is to correct the 
material failure. 

FIG. 3 depicts the algorithm used to process image data files from 
a database and identifies material failures if they have occurred. Two separate 
data files, scene (1) 14 and scene (2) 15, are made available for comparison. Both 

1 0 data files contain the same scene content, but they typically record images taken at 
different times. That is, the time between capturing the two images differs by a 
time At. Both image files or scenes undergo the process of orthorectification 16, 
that is, compensation for variations in position and angle at the time the scenes 
were recorded. This process is performed in order to allow an exact pixel by pixel 

1 5 comparison of the elements of a scene or image. It may or may not be necessary 
to correct the data in each scene for differences in the illumination 1 7 at the time 
each scene was recorded. Changes in the scene are identified in block 1 8 are used 
by the control computer 3 1 by detecting, using software, differences in the pixel 
content of the two scenes to be compared. Such changes may be reflected in the 

20 intensity of the pixels, or in the shape of an object, corresponding to a finite 

collection of pixels. Such methods for identification of pixel or object changes are 
well known to those skilled in the art. On the basis of such pixel changes the 
material failure type is identified in block 1 9 and the customer 1 3 is notified 20 of 
the existence of the failure. 

25 FIG. 4 illustrates a somewhat different embodiment of the present 

invention that uses an agent to aid in detecting material failures in a man-made 
object. An aerial platform 21 performs image capture 22 of a man-made structure 
(in this case a roadway, 23) in the manner previously described. FIG. 4 also 
shows image capture of a roadway 23 with an isolated man-made failure 24. In 

30 this instance, the detection of the man-made material failure is enhanced by the 
presence of a chemical or physical image contrast-enhancing agent. The contrast 



-6- 

agent for example, could be released to the environment as a consequence of the 
material failure occurring. For example, encapsulated fluorescent dyes, either as 
isolated molecular species or in crystalline form, embedded in a roadway could be 
released with crack formation. Triboluminescent materials that emit light upon 
5 mechanical impact could be sensed to determine the likelihood of material failure 
as a result of mechanical impact. Yellow-emitting halophosphate phosphors are 
well established as sensors when used in this mode of operation. In another use 
scenario, the contrast agent is incorporated into a man-made structure in a form 
where it becomes activated upon the occurrence of a material failure. For 

1 0 example, certain chemical species may undergo a color change upon exposure to 
air or other chemical species. An example is the use of corrosion-sensitive paints 
applied to pipelines. Color-changing chemical compounds (indicators) such as 
phenolphthalein or bromothymol blue, or fluorescing chemical compounds such 
as coumarins, can be incorporated into acrylic paints and applied to pipelines. 

1 5 Corrosion of the pipe causes a pH change in the paint; the consequence of which 
is a color change in the indicator or fluorescence. Such color changes are 
recorded remotely. Color-change materials exist that are sensitive to changes in 
pH, oxygen concentration, and the presence of trace quantities of various metal 
ions in the environment (chelating agents). The use of such materials is well 

20 known to those skilled in the art, and may be used singly or in combination, in 
remote sensing applications. 

Such materials in combination with the optical filter 3 in FIG. 1 
significantly improve the signal-to-noise ratio, and hence the detectability for 
remote sensing. The combination of the optical filter transmission function and 

25 the color-change to be detected as a result of a material failure will be optimized 
for each application. This optimization is well known to those skilled in the art. 

FIG 5 illustrates the capturing of an image, analysis to identify a 
material failure and communication over a computer network to deliver 
information to and receive payment from the customer 13. A satellite 25 or an 

30 aerial platform 26 captures an image of a scene 27 that contains a man-made 
structure (in this case electrical utility lines) to be analyzed. The image data is 



transmitted to a ground station 28 and transferred to the service provider's 
computer system 29. The image data is analyzed as previously described to 
determine whether a material failure has occurred. If a failure is detected the 
customer 13 for the service receives notification of the failure via a channel for 
5 example, a computer network such as the Internet, or via other means, such as 
telephony. The customer computer 30 receives the notification directly from over 
the computer network. The customer 13 subscribes to the service and pays for the 
service via the computer network. In this manner, the timely delivery of 
information regarding the status of a failure can be transmitted to the customer 

1 0 and the quality of service can be assured to be at a sufficiently high level. 

The invention has been described in detail with particular reference 
to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. For 
example, the control computer 31 can itself be reprogrammed from a remote 

15 location and would include all the necessary communication links to permit such 
reprogramming. 
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PARTS LIST 

1 sensor system 

2 capture device 

3 optical filter 

4 capture control circuit 

5 position and orientation control circuit 

6 image storage 

7 image processing circuit 

9 block 

10 scene database 

1 1 image analysis 

12 image analysis 

1 3 customer 

14 scene 

1 5 scene 

16 orthorectifi cation 

17 illumination 

18 block 

19 block 

20 notification 

21 aerial platform 

22 capture 

23 roadway 

24 man-made failure 

25 satellite 

26 aerial platform 

27 scene 

28 ground station 

29 service provider's computer system 

30 customer computer 

3 1 control computer 
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WHAT IS CLAIMED IS: 

1 . A method for capturing images of ground locations and for 
detecting the presence of material failure(s) or failures in man-made structures in 
such ground locations comprising the steps of: 

(a) providing an image sensor spaced remotely from the ground 
and which sequentially captures a number of images of various ground locations 
to provide digital images; 

(b) processing captured digital images to determine the 
presence of a potential material failure in a man-made structure in accordance 
with predetermined coordinate positions which locate the man-made structures in 
one or more of the captured digital images; and 

(c) indicating to a customer that a potential material failure has 
been detected in a predetermined coordinate position. 

2. The method of claim 1 further including: 

(d) sending captured processed digital images with detected 
potential material failures to a customer. 

3. The method of claim 1 wherein the digital image processing 
includes comparing previously captured digital images with newly captured 
digital images to determine variations in the captured digital images at the 
predetermined coordinates which indicate a potential material failure in a man- 
made structure. 

4. The method according to claim 1 wherein the digital images 
are captured by a capture device which is located in a fixed structure position 
above the ground location or in a moving structure such as an aircraft or satellite. 

5. The method of claim 3 wherein the image processing 
includes storing in memory a representation of different material failures to be 
detected and comparing the captured digital image with the material failures to 
determine the presence of a material failure, type of material failures and location 
of the material failures. 



- 10- 

6. A method for capturing images of ground locations and for 
detecting the presence of material failure(s) or failures in man-made structures 
having a detectable chemical agent in such ground locations comprising the steps 
of: 

(a) providing an image sensor spaced remotely from the ground 
and which sequentially captures a number of images of various ground locations 
to provide digital images; 

(b) processing captured digital images to determine changes in 
the chemical agent which indicate the presence of a potential material failure in a 
man-made structure in accordance with predetermined coordinate positions which 
locate the man-made structures in one or more of the captured digital images; and 

(c) indicating to a customer that a potential material failure has 
been detected in a predetermined coordinate position. 

7. The method of claim 6 wherein the chemical agent includes 
materials which when leaked from a receptacle are adapted to be detected. 

8. The method of claim 6 wherein the chemical agent includes 
materials which when released react with substances in the ground to provide a 
detectable material failure to the image sensor. 

9. A method for capturing images of ground locations and for 
detecting the presence of material failure(s) or failures in man-made structures in 
such ground locations comprising the steps of: 

(a) providing an image sensor spaced remotely from the ground 
and which sequentially captures a number of images of various ground locations 
to provide digital images; 

(b) processing captured digital images to determine the 
presence of a potential material failure in a man-made structure in accordance 
with predetermined coordinate positions which locate the man-made structures in 
one or more of the captured digital images; 

(c) indicating to a customer that a potential material failure has 
been detected in a predetermined coordinate position; and 

(d) correcting material failures. 
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1 0. A method for capturing images of ground locations and for 
detecting the presence of failure(s) or material failures in man-made structures in 
such ground locations and making payment for the detection or correction of 
detected material failures comprising the steps of: 

(a) providing an image sensor spaced remotely from the ground 
and which sequentially captures a number of images of various ground locations 
to provide digital images; 

(b) processing captured digital images to determine the 
presence of a potential material failure in a man-made structure in accordance 
with predetermined coordinate positions which locate the man-made structures in 
one or more of the captured digital images; 

(c) indicating to a customer that a potential material failure has 
been detected in a predetermined coordinate position; 

(d) correcting material failures; and 

(e) making payment for the material failure. 

1 1 . The method of claim 1 0 further including providing a 
chemical agent that includes which materials which when released reacts with 
substances in the ground to provide a detectable material failure to the image 
sensor. 

12. The method of claim 1 0 wherein the image processing 
includes comparing previously captured images with newly captured images to 
determine variations in a ground condition which could contain the material 
failure. 



0 
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ABSTRACT 

A method for capturing images of ground locations and for 
detecting the presence of failure(s) or material failures in man-made structures in 
such ground locations is disclosed. The method provides an image sensor spaced 
5 remotely from the ground and which sequentially captures a number of images of 
various ground locations to provide digital images; processing captured digital 
images to determine the presence of a potential material failure in a man-made 
structure in accordance with predetermined coordinate positions which locate the 
man-made structures in one or more of the captured digital images; and indicating 
10 to a customer that a potential material failure has been detected in a predetermined 
coordinate position. 
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